robots.txt files of Alexa's top 10k websites: 2012-10
There Is No Preview Available For This Item
This item does not appear to have any files that can be experienced on Archive.org.
Please download files in this item to interact with them on your computer.
Show all files
A daily download of robots.txt files from the top 10k websites according to Alexa.
This is the collected files from 2012-10.
- aria2c-DATE.log.gz are the logs from the download tool.
- files-DATE.7z are the robots.txt files in a a/abc.com/DATE structure.
- removal-DATE.log.gz are logs of what files were deleted prior to 7zipping as they were thought to be no robots.txt files.
- top-1m.csv-DATE.zip are the Alexa 1m toplists.
Uploaded by o373Nmu6Mf on